This is an evaluation of forecasts for Covid-19 case and death numbers in 32 European countries submitted to the European COVID-19 Forecast Hub. You can find more information on the European Forecast Hub Github page.

This report is intended as a basic evaluation of forecasts that helps modellers to better understand their performance. The structure and visualisations are likely subject to change in the future and we cannot rule out any mistakes. If you have questions or want to give feedback, please create an issue on our github repository. Note that all forecast dates have been changed to the corresponding submission date (every Monday) to allow easier comparison.


Forecast accuracy

Here is an overview of different evaluation metrics. See below for a more detailed explanation of the scoring metrics used. ‘Overall’ shows scores for all past weeks, ‘latest’ only spans the last 5-6 weeks of data. ‘Detailed’ represents the full data set that you can download for your own analysis.

Austria

overall

latest

Cases

Deaths

overall by horizon

Detailed

Belgium

overall

latest

Cases

Deaths

overall by horizon

Detailed

Bulgaria

overall

latest

Cases

Deaths

overall by horizon

Detailed

Croatia

overall

latest

Cases

Deaths

overall by horizon

Detailed

Cyprus

overall

latest

Cases

Deaths

overall by horizon

Detailed

Czechia

overall

latest

Cases

Deaths

overall by horizon

Detailed

Denmark

overall

latest

Cases

Deaths

overall by horizon

Detailed

Estonia

overall

latest

Cases

Deaths

overall by horizon

Detailed

Finland

overall

latest

Cases

Deaths

overall by horizon

Detailed

France

overall

latest

Cases

Deaths

overall by horizon

Detailed

Germany

overall

latest

Cases

Deaths

overall by horizon

Detailed

Greece

overall

latest

Cases

Deaths

overall by horizon

Detailed

Hungary

overall

latest

Cases

Deaths

overall by horizon

Detailed

Iceland

overall

latest

Cases

Deaths

overall by horizon

Detailed

Ireland

overall

latest

Cases

Deaths

overall by horizon

Detailed

Italy

overall

latest

Cases

Deaths

overall by horizon

Detailed

Latvia

overall

latest

Cases

Deaths

overall by horizon

Detailed

Liechtenstein

overall

latest

Cases

Deaths

overall by horizon

Detailed

Lithuania

overall

latest

Cases

Deaths

overall by horizon

Detailed

Luxembourg

overall

latest

Cases

Deaths

overall by horizon

Detailed

Malta

overall

latest

Cases

Deaths

overall by horizon

Detailed

Netherlands

overall

latest

Cases

Deaths

overall by horizon

Detailed

Norway

overall

latest

Cases

Deaths

overall by horizon

Detailed

Poland

overall

latest

Cases

Deaths

overall by horizon

Detailed

Portugal

overall

latest

Cases

Deaths

overall by horizon

Detailed

Romania

overall

latest

Cases

Deaths

overall by horizon

Detailed

Slovakia

overall

latest

Cases

Deaths

overall by horizon

Detailed

Slovenia

overall

latest

Cases

Deaths

overall by horizon

Detailed

Spain

overall

latest

Cases

Deaths

overall by horizon

Detailed

Sweden

overall

latest

Cases

Deaths

overall by horizon

Detailed

Switzerland

overall

latest

Cases

Deaths

overall by horizon

Detailed

United Kingdom

overall

latest

Cases

Deaths

overall by horizon

Detailed


Evaluation metrics


Scores over time

Here you can see a visualisation of forecaster scores over time. The first tab shows the weighted interval score. Other tabs show the components of the interval score, sharpness (how narrow are forecasts - smaller is better), and penalties for underprediction and overprediction.

Austria

Weighted interval score

Overprediction

Underprediction

Sharpness

Belgium

Weighted interval score

Overprediction

Underprediction

Sharpness

Bulgaria

Weighted interval score

Overprediction

Underprediction

Sharpness

Croatia

Weighted interval score

Overprediction

Underprediction

Sharpness

Cyprus

Weighted interval score

Overprediction

Underprediction

Sharpness

Czechia

Weighted interval score

Overprediction

Underprediction

Sharpness

Denmark

Weighted interval score

Overprediction

Underprediction

Sharpness

Estonia

Weighted interval score

Overprediction

Underprediction

Sharpness

Finland

Weighted interval score

Overprediction

Underprediction

Sharpness

France

Weighted interval score

Overprediction

Underprediction

Sharpness

Germany

Weighted interval score

Overprediction

Underprediction

Sharpness

Greece

Weighted interval score

Overprediction

Underprediction

Sharpness

Hungary

Weighted interval score

Overprediction

Underprediction

Sharpness

Iceland

Weighted interval score

Overprediction

Underprediction

Sharpness

Ireland

Weighted interval score

Overprediction

Underprediction

Sharpness

Italy

Weighted interval score

Overprediction

Underprediction

Sharpness

Latvia

Weighted interval score

Overprediction

Underprediction

Sharpness

Liechtenstein

Weighted interval score

Overprediction

Underprediction

Sharpness

Lithuania

Weighted interval score

Overprediction

Underprediction

Sharpness

Luxembourg

Weighted interval score

Overprediction

Underprediction

Sharpness

Malta

Weighted interval score

Overprediction

Underprediction

Sharpness

Netherlands

Weighted interval score

Overprediction

Underprediction

Sharpness

Norway

Weighted interval score

Overprediction

Underprediction

Sharpness

Poland

Weighted interval score

Overprediction

Underprediction

Sharpness

Portugal

Weighted interval score

Overprediction

Underprediction

Sharpness

Romania

Weighted interval score

Overprediction

Underprediction

Sharpness

Slovakia

Weighted interval score

Overprediction

Underprediction

Sharpness

Slovenia

Weighted interval score

Overprediction

Underprediction

Sharpness

Spain

Weighted interval score

Overprediction

Underprediction

Sharpness

Sweden

Weighted interval score

Overprediction

Underprediction

Sharpness

Switzerland

Weighted interval score

Overprediction

Underprediction

Sharpness

United Kingdom

Weighted interval score

Overprediction

Underprediction

Sharpness


WIS decomposition

As mentionend above, the weighted interval score can be decomposed into three parts: sharpness (the amount of uncertainty around the forecast), overprediction and underprediction. This visualisation gives an impression of the distribution between these three forms of penalties for the different models.

overall

Austria

Belgium

Bulgaria

Croatia

Cyprus

Czechia

Denmark

Estonia

Finland

France

Germany

Greece

Hungary

Iceland

Ireland

Italy

Latvia

Liechtenstein

Lithuania

Luxembourg

Malta

Netherlands

Norway

Poland

Portugal

Romania

Slovakia

Slovenia

Spain

Sweden

Switzerland

United Kingdom

If you want to learn more about a model, you can go the the ‘data-processed’-folder of the European Forecast Hub github repository, select a model and access the metadata file with further information provided by the model authors.